AUTOMORPHISMS OF HYPERELLIPTIC MODULAR CURVES Xo{N) IN 
POSITIVE CHARACTERISTIC 

ARISTIDES KONTOGEORGIS AND YIFAN YANG 



Abstract. We study the automorphism groups of the reduction Xq(N) x ¥p of a mod- 
ular curve X() {N) over primes p \ N. 



1. Introduction 

Let ^ S he a family of curves of genus g > 2 over a base scheme S. For every 
point P : Spec fc ^ 5, we will consider the absolute automorphism group of the fibre P 
to be the automorphism group Autj.(^ X5 Specfc) where k is the algebraic closure of 
k. Any automorphism cr acts like the identity on k, so in our setting there is no Gal(fc/fc) 
contribution to the automorphism group of any special fibre. The following theorem due to 
P. Deligne, D. Mumford Jl] lemma 1. 12] compares the automorphism groups of the generic 
and special fibres: 

Proposition 1.1 ({5}). Consider a stable curve ^/ ~ ^ /S* over a scheme S and let denote 
its generic fibre. Every automorphism (j) : ^t-i can be extended to an automorphism 

(j): ^ ^ 3^. 

Of course the special fibre of might possess automorphisms that can not be Ufted to 
the generic fibre. For example the Fermat curve 

can be considered as a stable curve over Spec Z[pq-j-] and has automorphism group (/x„ x 
Mn) S3 for all geometric fibres above the primes q ^ p but PGU(3,p^^) for the prime 
p USD, ini- A special fibre JTp j:' Xs S/p with Aut( JT^) > Aut(^^) will be called 
exceptional. In general we know that there are finite many exceptional fibres and it is an 
interesting problem to determine exactly the exceptional fibres. 

There are some results towards this problem for some curves of arithmetic interest. 
A. Adler HI and C.S. Rajan ||2T] proved for the modular curves X{N), that X{l\)j, := 
X{\\) XspocZ SpecFs has the Mathieu group Mn as the full automorphism group. C. 
Ritzenthaler in ll22l and P. Bending, A. Carmina, R. Guralnick [Si studied the automor- 
phism groups of the reductions X{q)p of modular curves X{q) for various primes p. It 
turns out that the reduction ^(7)3 of X{7) at the prime 3 has an automorphism group 
PGU(3, 3), and X(7)3 and X(ll)3 are the only cases where AutX(q)p > AMtX{q) = 
PSL(2,p). 

In this paper we will investigate some modular curves of the form Xi^{N). Igusa ||T3l 
proved that Xq (N) has a non-singular projective model which is defined by equations over 
Q whose reduction modulo primes p,p\ N are also non-singular, or in a more abstract lan- 
guage that there is a proper smooth curve (N) — *■ Z[l/A^] so that for p £ Spec Z[l/iV] 
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the reduction ^o{N) xgpocz Fp is the moduli space of elUptic curves with a fixed cycUc 
subgroup of order N. 

The automorphism group of the curve Xo{N) at the generic fibre is now well under- 
stood. Let us call an automorphism a of Xq (TV) modular if a arises from the normalizer 
of ro(A^) in PGLj(Q). Then the results of |Igl[Il[l9l can be summarized as follows. 

Theorem 1.2 (|l6l[H[l9l). Assume diat Xq{N) has genus g > 2. Then all the automor- 
phisms of Xq{N) are modular, with only two exceptions, namely the cases N = 37,63, 
which have extra non-modular involutions. 

Remark 1.3. Note that if 4 | and % \ N, then all modular automorphisms are of the 
Atkin-Lehner type 121 Theorem 8] . In particular, if 4 f and 9 f A^, then the automorphism 
group is an elementary abelian 2-group. 

Denote by A{N^O) the absolute automorphism group of ^o{N) at the generic fibre 
and by A{N,p) the absolute automorphism group at the reduction at the prime p. The 
problem of determining the exact primes for which A{N,p) > A{N, 0) seems difficult in 
general. However, when Xo{N) is hyperelliptic, the situation is relatively simple because 
the function field of a hyperelliptic curve has a unique genus zero subfield of degree 2, and 
the problem of determining the automorphism group of a hyperelliptic curve is essentially 
the same as that of determining the automorphisms of a projective line that permute a 
set of marked points. Therefore, as a starting point for our general problem of studying 
automorphisms of Xo{N) in positive characteristics, in this note, we will investigate the 
automorphism groups of hyperelliptic modular curves Xo{N). 

Theorem 1.4. Table\l\is the complete list of integers N and primes p such that the reduc- 
tion of the hyperelliptic modular curve Xo{N) modulo p has exceptional automorphisms. 

Here in the table, the notation Z3„ denotes the dihedral group of order 2n, 

A (a,6,c I ba^^^^i^-i 5-1^3^^-1 ^^^-i^^-i^-i^^-i^^ (a^ife-^cfe^i)^), 
is a group of order 672, 

B :~ {a, b, c \ c? , a~^, b~^a~^ba, {cb~^)^ , a~^bca^cac). 
is a group of order 240, and 

is a group of order An. Moreover, in the case of Xo(37) in characteristic 2, the notation 
£"32- represents the extraspecial group -E32- = (-D4 x Qg,)/{{a, b)), where a and b denote 
the nontrivial elements of the centers of and Qg, respectively. 

Acknowledgments: This paper was completed during the authors' visit at Max-Planck 
Institut fiir Mathematik in Bonn. The authors would like to thank the institute for its 
support and hospitality. 

2. Automorphisms in characteristic p^2 

According to ||20| . there are exactly 19 values of N such that Xo(Af) is hyperelliptic. 
The equations of the form y^ = f{x) for hyperelliptic Xq{N) have been computed by 
several authors El IH |28l . They are tabulated in Table [21 along with their modular auto- 
morphisms. Here, for a divisor e of iV with (e, N/e) = 1, we let Wd be the Atkin-Lehner 
involution corresponding to the normalizer 

{Tn de)' <^de-bcN/e = \, 
2 



Table 1 . Automorphism groups of Xo{N) in positive characteristics 



N 


Genus 


Generic Aut 


Exceptional primes 


Exceptional Aut 


22 


2 


(Z/2Z)2 


3,29 
101 




23 


2 


Z/2Z 


3,13,29,43,101,5623 


(Z/2Z)2 


26 


2 


(Z/2Z)2 


7,31 
41,89 


^6 


28 


2 


^6 


3 

5 
11 


GT/S 3") 

^6 


29 


2 


Z/2Z 


19 

5,67,137,51241 


(Z/2Z)2 


30 


3 


(Z/2Z)3 


23 




31 


2 


Z/2Z 


3 

5,11,37,67,131,149 


(Z/2Z)2 


33 


3 




2 

19 

47 


GL(2,2) X (Z/2Z) 
(Z/2Z) X (Z/4Z) 
(Z/2Z)3 


35 


3 


{Z/2'Lf 


— 


— 


37 


2 


(Z/2Z)2 


2 

7,31 
on /^i 


E:i2- (Z/5Z) 

^6 

V4 


oy 


Q 
O 


yiLj liL ) 




29 


yiLj LiL j 

(Z/2Z) X (Z/4Z) 


40 


3 


(Z/2Z) X D4 


3 


Vs 


41 


3 


Z/2Z 


17 


(Z/2Z)2 


46 


5 


(Z/2Z)2 


3 


(Z/2Z) X (Z/4Z) 


47 


4 


Z/2Z 






48 


3 


(Z/2Z) X 5*4 


7 


A |A| = 672 


50 


2 


(Z/2Z)2 


3 

37 


^6 


59 


5 


Z/2Z 






71 


6 


Z/2Z 







of ro(A^). For Xo(28) and Xo(40), the notation Wi/2 represents the automorphism coming 
from the normalizer 

'1 1/2^ 

The notation Wi/4 in the case Xo(48) carries a similar meaning. 

These models are not the Igusa curves. They have a singularity at infinity but are smooth 
otherwise. The automorphism group of a non-singular curve is a birational invariant and 
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Table 2. List of hyperelliptic curves Xq {N) 



N 


Equation/Automorphisms 


22 


= (x^ + ix^ + 8x + 4)(x^ + 8x^ + 16a: + 16) 
^2 (x, y) ^ (4,/x, Sy/x^), TOii : (x,y) ^ (x. ~y) 


23 


y^ = (x^ - X + l)(x^ - Sx^ + 3a: - 7) 
""23 ■ (a:, y) " (x, -y) 


26 


y^ = x'^ - Sx^ + 8x* - 18a:^ + Sx^ - 8a: + 1 

™13 : V) " (1/=:. y/x^), 11126 • (^i !/) "!/) 


28 


= (a:^ + 7)(a:2 + a: + 2)(a:2 - a: + 2) 
^4 : (a:, y) ^ ((a: + 3)/(a: - 1), 8j//(a: - 1)^), 1117 : (x,y) (a:, -;;), ii>i/2 • (^,y) ^ (-a:, -;;) 


29 


y^ — x^ - ix^ — 12a!* + 2x^ + Sx^ + Sx - 7 
■ V) ^ (2:, -h) 


30 


= (a:^ + Ax - l){x'^ + x - l)(x* + a:^ + 2x'^ - a: + 1) 
ii>2 : (x, y) M ((a: + l)/(a: - 1), -4a/(a: - 1)*), 11)3 : (a:, y) h-. ( - 1/a:, -y/x*), 
™15 : (x, y) " (x, -a) 


31 


= (x^ - 6x2 _ 51 _ l){x^ - 2x2 _ ^ ^ 

^31 ■ y) ^ -y) 


33 


y2 = (x2 + X + 3)(x'' + 7x^ + 28x* + 59x^ + 84x2 _,_ ^ 37) 
^3 : y) " (3/x, — gy/x"), iDn : (x, y) i-> (x, -j/) 


35 


1/2 = (x2 + X - l){x'' - 5x'' - 9x3 _ 5^ _ 

u:? : (x, h) m ( — 1/x, h/x*), 1/135 • (2:, a) ^ (x, — y) 


37 


y^ ^ x^ + 14x^ + 35x* + 48x3 + 35x2 _|_ ^ j 
^37 : (x,!/) ^ (I/x.b/x^) 


39 


1/2 = (a;* _ 7a;3 _|_ 113.2 _ 73. ^ l)(x'' + x^ - x2 + x + 1) 
^3 : y) ^ (1/2:. h/x*), ™39 : (x, J/) h-> (x, —1/) 


40 


y^ = x^ + 8x^ - 2x* + 8x2 ^ ^ 

: (x, //) ^ (-1/x, -t//x*), wg : (x, //) ^ ((1 - x)/(l + x), -4y/(x + 1)*) 
™l/2 : m) " (-^. m) 


41 


y^ = x^ - ix'' - 8x^ + lOx^ + 20x'' + 8x3 _ 15^.2 _ _ g 
Wii ■■ (x, !/) (x, -y) 


46 


y2 = (x3 + a;2 _|_ 2x + l)(x3 + 4x2 + 43. + 8){x° + 5x^ + 14x* + 25x3 + 28x2 + 20x + 8) 
"'23 : (x,y) 1-* (x, —y), : (x,y) i-» (2/x, 8h/x°) 


47 


y2 = (x^ + 4x'' + 7x3 ^ g^2 _|_ 4^ _|_ i)(j.5 _ 5^3 _ 20a;2 _ 24x - 19) 
"47 ■ (3:, y) (x, -y) 


48 


y2 = (x* - 2x3 _,_ 2j;2 + 2x + l)(x* + 2x3 _,_ _ 2x + 1) = x'^ + 14x* + 1 
™l/4 (x,y) —> (ix,y), i/)3 : (x, y) ( - 1/x, — y/x*) , 
"16 : (a:, y) ((1 - x)/(l + x), -4y/(l + x)*) 


50 


y2 = x'' - 4x^ - 10x3 - 4x + 1 

W2 ■■ (x, y) ^ (1/x, y/x3), U150 : (x, y) m (x, — y) 


59 


y2 = (x3 + 2x2 ^ ij(.^9 ^ 2x'^ - 4x'' - 21x'^ - 44x'' - 60x* - 61x3 _ 4g^2 _ 24^ _ 
"sg : (x, y) (x, -y) 


71 


y2 = (j,''' _ 3a;S _|_ 2x'' + x* - 2x3 ^ 2x2 _ j. + i)(3,7 _ 7^6 _(_ ^4^5 _ ii^4 _(_ ^4^3 _ 14^2 _ ^ _ 7) 
wji ■■ (x, y) (x, -y) 



one can work with singular models of the corresponding function fields. But one has to be 
careful if he wants to compute reductions out of the singular model. 

Proposition 2.1. Lety^ = f(x) where f(x) e Z[x] is a polynomial with integer coefficient 
such that dcg(/) = mod 2. 

Suppose that f{x) has no multiple roots and denote pi, . . . , ps G Q the set of roots of 
f. Let A Cz Z be the discriminant of the polynomial f{x). Then, the relative curve 
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is a smooth family. The set of mots {pi\i=i....^s gives rise to horizontal divisors pi that 
intersect the generic fibre ofW at pi. The intersections of pi with the fibre 'Wp at the prime 
p are roots of the polynomial f mod p. 

Let ^ Z [(2iV)^^] be the Igusa family (with the 2-fiber removed if 2 \ N). For 
every p such that (p, A) = 1 the function fields of the curves <^ and have the same 
automorphism group. 

Proof. To prove that the map given in eq. ([T]i is smooth we will show that is smooth 
for every p so that p\ A. Set F{x) ~ — f{x). We compute 



Therefore, a nonsmooth point appears at y = (since p ^ 2) and at a double root of 
f{x) mod p, i.e., only at primes dividing the discriminant. 

The Igusa family has good reduction at the prime 2 if 2 f A^. The hyperelliptic models 
in Table |2]have always bad reduction at the prime 2. Therefore the reduction of the curve 

is not related to the Igusa curve. In the 2 \ N case the special fibre of the Igusa model 
at the prime 2, is a nonsingular curve and the cover ^ (^)2 —* Pf. is given in terms of an 
Artin-Schreier extension. The automorphisms in characteristic 2 will be treated separately 
in Section|5] 

The Igusa curve is the normalization of at infinity at every prime in SpccZ[A^^]. 
It is known that every point P in the generic fibre defines a unique thickening, i.e., a 
horizontal branch divisor intersecting the generic fibre at P. Since deg(/) = mod 2 the 
place at infinity is not ramified at the cover Xi^{N) — ^ P^. Therefore the set of branch 
points of the cover Xo(A^) is contained in the generic fibre of . Moreover the 

thickenings pi of each point pi in (g)Q Q C <^ Q do not intersect the singular locus 
of the projective closure of ^ . 

The automorphism group of is equal to the automorphism group of the correspond- 
ing function field and ^p, 'S/p share the same function field. □ 

The automorphism group of hyperelliptic curves is a well studied object Q, lfT2l . ll25l . 
There is the hyperelliptic involution j so that (7) is a normal subgroup of the whole auto- 
morphism group, and moreover Xj (j) is isomorphic to the projective line. The quotient 
G = Aut{X) / {j) is called the reduced group. The reduced group is a finite subgroup of 
the group of automorphisms of the projective line and these groups are given by proposi- 
tion [Zj] We introduce the following notation: For every w G Aut{X) we will denote by 
w the image of w in the reduced group G := Aut{X)/{j). 

Consider the cover X X/{j) = Pj,. Let Pi, . . . , € Fl be the branched points of 
the cover X — > P[.. We will call the set Pi , . . . , P, the branched hyperelliptic locus. 

We know that the reduced group G induces a permutation action on them. If the points 
Pi , . . . , P5 are in general position then there is no finite subgroup of PGL(2, k) permuting 
them and Aut{X) ~ (j). The existence of additional automorphisms is a matter of special 
configurations of points in the configuration space of s points in the projective line. The 
problem of determining the primes p, so that the reduction of a hyperelliptic curve at that 
primes has more automorphisms, is then reduced to the problem of determining the primes 
at which the branch hyperelliptic locus becomes more symmetric. 

We now return to the theory of hyperelliptic curves Xq (N). If the number N is compos- 
ite or TV = 37 then the group of automorphisms at the generic fibre is bigger than Z/2Z. 
This means that we have non trivial reduced group at the generic fibre. This situation gives 



dF 



d£ dl 

dx dx 



dy 



22/, 
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us a lot of information for the location of fixed points of any extra automorphism in the 
reduction modulo p and on the possible extra automorphism groups. 

For instance the case of hyperelliptic curves that admit an extra involution is studied by 
T. Shaska and J. Gutierrez ll25l . lfT2l . They introduced the theory of dihedral invariants, 
a theory that allows us to compute every possible extra automorphism for hyperelliptic 
curves of genera 2, 3, and also gives us a lot of information in the bigger genus cases. 

If the reduced automorphism group at the generic fibre is trivial then we use a brute 
force method in order to compute any extra automorphism group. This is a demanding 
computational problem that needs several days of processing time. 

Once the reduced group G of a hyperelliptic curve is determined the group G is given 
in terms of an extension of groups 

1 ^ (j) ^ G ^ G ^ 1. 

For a cohomological approach to the structure of G we refer to ifTSl . It is known the group 
structure of G depends on whether the fixed points of G are in the branch locus of the cover 
Xo(7V)^ Xo(iV)<J> =P^l4l. 

Proposition 2.2. Let k be an algebraically closed field of characteristic p > 0. Let G be 
a finite subgroup o/Aut(P^) — PGL(2, k), let Pi, . . . ,P,. denote the number of branch 
points in the cover P^. Denote the ramification degree of Q Pi by e^. Then G is 

one of the following groups: 

(1) Cyclic group 'L/n'L of order n relatively prime to p with r = 2, and ei ~ 62 = n- 

(2) Elementary abelian p-group with r = 1, ei = \G\. 

(3) Dihedral group Dn of order 2n, with p = 2, {p, n) ~ I, r ~ 2, ei — 2, 62 — n, 
or p ^ 2, (p, n) = 1, r = 3, ei = 62 = 2, 63 = n. 

(4) Alternating group ^4 with p ^ 2, 3, r = 3, ei = 2, 62 = £3 = 3 

(5) Symmetric group with p 7^ 2, 3, r = 3, ei = 2, 62 = 3, 63 = 4. 

(6) Alternating group A<^ with p — 3, r — 2, ei — 6, 62 ^ 5, or p 2,3,5 r = 3, 
ei = 2, 62 = 3, 63 = 5. 

(7) Semidirect product of an elementary abelian p-group of order with a cyclic 
group Z/nZ of order n with — 1), r = 2, ei = |G|, 62 = n. 

(8) PSL(2,p*) with p^2,r = 2,ei= p^p* - l)/2, 62 = (p* + l)/2. 

(9) PGL(2,p*) with r = 2 ei = p*(p* - 1), 62 = p* + 1. 

Proof [30, Th. 1] □ 

Remark 2.3. Observe that the groups A4, Si, PSL(2,p*), PGL(2,p*) contain a dihe- 
dral group Dn, for {n,p) = 1. 

Let f{x) be a polynomial of degree s with roots pi, . . . , p^. A simple computation 
shows that 

Lemma 2.4. Let A = ^ ^ be an invertible matrix. If c ~ and pic ^ CLfor all roots 
of f then: 

\cx + dj (cx + d)'' i=i\ -ptC + a 

If c = then pic 7^ a since A is invertible and 
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If there is some pi^ such that cpi ~ a then c ^ and 



'PiC 



Definition 2.5. Let A denote the Mobius transformation x i-^- [ax + h)/{cx + d). We will 
denote by f\ the polynomial 

n 

. \ -PiC + a 

t—l,PiC^a 

Notice that deg(/) = deg(/^) if cpi ^ a for all roots pi of /, or dcg(/) = dcg(/J^) + 1 
if there is a root pi such that cpi — a. 

Lemma 2.6. Let A be a Mobius transformation and set s = deg /. If s = mod 2 then 
the curves y'^ = f{x) and = fXi^) '^''^ isomorphic, over a quadratic extension ofQ. 

Proof. The two curves become isomorphic under the change of variables 

'ax + b yC^'"^ 



^^'^^ ■ ' \cx + d' {cx + dy/\ 
where C is a constant depending on which of the cases in (|2]i,(l3]i,(|4l) we are. □ 

If f{x) — J2t=o 0-1^^"^ is a polynomial with aQ ^ then we will denote by f *{x) the 
rec/procaZ polynomial given by = Qq^x'^ f{l/x) = flg^^ X]t=o Q^i^s^*"''- 
If 2|s and /(O) ^ then the hyperelliptic curves 

y^^fix) andy^^f*{x) 

are isomorphic. 

Lemma 2.7. We will denote by a the automorphism o/P^ sending x to —x. Consider the 
hyperelliptic curve 

s 

that has a in the reduced automorphism group. There is a cyclic group (a) < Cd < 
PGL(2, k) that is a subgroup of the reduced automorphism group of X if and only if d\s 
and as = Ofor all d\6. 

3. Curves with non trivial reduced group 

3.1. Curves of Genus 2 with an extra involution. In this case we consider the set of 
curves Xo{N) that are of genus 2 and have reduced group isomorphic to Z/2Z. This is the 
case for N = 22, 26, 28, 37, 50. 

For genus two curves with reduced group Z/2Z there is a well developed theory due 
to Guttierez, Shaska IIT2II . ll25l . namely the theory of dihedral invariants that reduce the 
computation of the tame part of the automorphism group to the computation of several 
invariants of the curve. 

Since Xq(N) has reduced group Z/2Z, we can find a model of the curve so that the 
generator a of the reduced group acts like x 1-^ —x. Thus, the model of our curve is of the 
form: 



(5) 



y2 ^ ^2g+2 ^^^2g _^ . . . + ^^^^ + I 
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Table 3 . Dihedral invariants of curves of genus 2 with extra involutions 







U2 


22 


-17322/14641 


130/121 


26 


-4351/2704 


15/13 


28 


43625/784 


125/7 


37 


-25642/1369 


-198/37 


50 


-135/16 


-5 



The dihedral invariants are then given by := af~''''^ai+a|~'+^ag-i+i fori 
In particular for a genus two curve the dihedral invariants are given by ui 
U2 = 2aia3. Let Vn denote the group 

In ll26l exam. 5.2], ll27l T. Shaska and H. Volklein proved that the automorphism group is 
isomorphic to 

(1) if and only if (ui, Us) = (0, 0) or (ui, us) = (6750,450) 

(2) (a) GL(2, 3) if and only if (ui , us) = (-250, 50) and p 7^ 5 
(b) B if andonly if (ui,U2) = (-250,50) andp 5 

(3) De if and only if - 220^2 - 16ui + 4500 = 0, 

(4) D4 if and only if 2ul - = for U2 ^ 2, 18, 0, 50, 450. 

(Cases 0, 450, 50 are reduced to Cases ([TJ and ^ ). The group B mentioned above is given 
by: 

B :— (a, b, c \ (? , , b^^a^^ba, {cb^^)^ , a^^bca^cac). 

Using the exact action of the generator of the reduced group given in Table |2] we can 
find a model of the form (|5]l by diagonalizing the 2x2 matrix representing the Mobius 
transformation. The dihedral invariants are then computed (O and this information allows 
us to compute the automorphism groups given in table [T] 

3.2. Curves of genus 3 with an extra involution. A similar approach to the genus 2 in 
terms of the dihedral invariants can be given for the hyperelliptic curves with an involu- 
tion in the reduced group that have genus 3. A complete list of all possible automorphism 
groups that can appear in the genus 3 case together with necessary conditions on the di- 
hedral invariants is given in the following theorem due to J.Gutierrez, D. Sevilla and T. 
Shaska ifTDl . 

Theorem 3.1. Consider a hyperelliptic curve X of genus 3 with an extra involution. Let 
G denote the full automorphism group Aut(X) of X and G the reduced automorphism 
group. If the curve has G, G as in the first two columns oftable^then the conditions given 
in the 3rd column of tableware satisfied, where 

Ei{ui,U2) 
E2{ui,U2) 



= 1,...,.9. 

= a\ + 0.2, 



588u2 - 5(u3 - 8)(9m3 - 1024), 

. 9 c, 873 2 149504 1048576 
7 Ml Wi Uo H u-i 

1 g 3 2 3^ g J g 



Table 4. Possible Automorphism groups in the 5 = 3 case. 



G 


G 


Conditions 


(Z/2Z)^ 


(Z/2Z)2 


2ui — ii3 — 


(Z/2Z) X Ds 


Ds 


2ui — U.3 — 0, ai — a3 


Vs 


Die 


2ui — ii3 — 0, ai — a2 — (13 — 


(Z/2Z) X S4 


Si 


(2ui — = 0, ai — 01 Ei {ul, U2) — E2 {ui, U2) — 0) and 
({ui,U2, u^) = (0, 196, 0) or {81ui , 27'U2, 8^3) ^ {8192, -1280, 128)) 




De 


Ei{ux,U2) ^ E2(ui,U2) ^ 


Z/2Z X Z/4Z 


Di 


2ui + ^ 




D12 


(2ui + 11,3 — Oor£;i{ul, U2) — E2{ux, U2) — 0) and 
■U2 - 0. 



Table 5. Models f{x) for N = 33, 35, 39, 40, 48. 



N 




30 


+ 


33 


+ 


35 


bx^ - 


39 


27 x^ 


40 


x^ - 


48 


x^ + 



(276+184 V2) g , (-184^/2+276) , 765+540 

(-540 x/2-765)-^ " (-540 V2- 765) (-540 \/2-765) 

(-240v/3+508)x° 4 (508+240 v/3):r^ 473+264^3 

- 264 \/3+473 - 264 \/3+473 - 264 V3+473 

I- (140 + 128 i) x^ - 34 + (140 - 128 i)x^ +5 
- 22 • 97 + 2 • 29 a;-* + 22 • 11 a:^ + 3 
18a;'* + 1 
14a;'* + 1 



and Vs, Uq are the groups with presentations: 

Vs := {x,y\x\v\{xy)\{x-^vf) 
Ue ■■= {x,y \ x^,y^^,xyxy'^). 

The curves Xq {N) that are hyperelliptic of genus 3 and have an involution in the re- 
duced group correspond to e {39, 40, 48, 33, 35, 30}. We compute first a hyperelliptic 
model of our curves so that the generator a of an extra involution is given by cr : a; i-^ —x. 
These models are given in table |5] 

The dihedral invariants in the case of genus 3 curves with an extra involution are given 
by Ul = af + a|, U2 = {of + 03)02, M3 = 20103, where 

y'^ = x^ + aix^ + 02a;'* + 03a;2 + 1 

is a normalized model of the hyperelliptic curve. The dihedral invariants for the hyperel- 
liptic curves Xo{N) of genus 3 with extra involution are given in table|6] 

For = 39 we see that possible exceptional primes are 5, 29, 181 and then by reducing 
the coefficients modulo each of these primes we see that ^4(39, 5) = (Z/2Z)'^, ^4(39, 29) = 
(Z/2Z) X (Z/4Z). The prime 181 is not exceptional. For A^ = 40 the possible exceptional 
primes are 3, 7 and A(40, 0) = (Z/2Z) x Ds and ^(40, 3) = Vs while for A(40, 7) ^ 
(Z/2Z) X 5*4. For A^ = 48 the possible exceptional primes is 7 and we have ^(48, 0) = 
(Z/2Z) X Si, A(48, 7) = Vs. The full group of ^0(48)7 was studied by the first author in 
lfT6l and is isomorphic to an extension of PGL(2, 7) by Z/2Z. Using magma we compute 
that this group admits the following presentation: 

A := (^a,b,c\ ,ba-%-^a-\b-^a^ba-\ba-^cb-^a-^ca-~'^c, [a^^^b-^cb-^f^ . 

9 



Table 6. Exceptional primes for N = 30, 33, 35, 39, 40, 48. 



N 


Factors of 


Factors of 


Possible excep- 


Exceptional 




2ui + ul 


2ui - ul 


tional primes 


primes 


30 


2,23,17 


2,17 


23,17 


17 


33 


2,19,31,103 


2,3,47 


19,31,47,103 


19,47 


35 


3,67 


2,7 


3,67 




39 


2,29,181 


2,5,13 


5,29,181 


5,29 


40 








3 


3 


48 








7 


7 



For = 30, 33, 35 the situation is a little more difficult, since the normal model is 
not defined over Z but in the principal ideal domains l\\f% , Z[-\/3] , respectively. We 
compute that the dihedral invariants in the cases N = 33, 35 are in Q and in table |6] we 
present the prime factors of the numerator For the = 30 case the dihedral invariants 
are in Q[a/2], and we compute that the principal ideal generated by the numerators of 
2uf-Ug and 2u\ + u1 have the prime ideals h, hs.,!, hd.,2 and h, hi,!, I23,2, h7,i, h7,2, 
where I2 = (2, V2)^^] and l23s,In,i are the prime ideals that extend the prime ideals 
23Z, 17Z of the ring of rational integers. Since both conjugate prime ideals hs.i, i = 1,2 
(resp. In.i) are divisors of 2ul — (resp. 2u^ + u^) we see that 17, 23|(2ui + u^) 
and 17|(2ui + Ug). The possible exceptional primes are p = 17,33. We reduce the 
coefficients modulo p ~ 17, 33 and we compute the automorphism group. It turns out that 
A(30, 17) ^ Vs, A(33, 19) (Z/2Z) x (Z/4Z), A(33, 47) ^ (Z/2Z)^l 

3.3. Elementary abelian groups. We will study now whether there might be an elemen- 
tary abelian subgroup E = {Z/pT^Y as a subgroup of the reduced group of a hyperelliptic 
curve. We are in the composite N case so there is already a cyclic group inside the reduced 
group. For every maximal cyclic subgroup of the reduced automorphism group we change 
coordinates so that the model of our curve is of the form: 

s 

= fix) = X! «2i/a;^''. 

i/=0 

If there is an elementary abelian group inside the reduced group modulo some prime p 
then the reduced group contains E xi {IjjmX), for some m prime to p. The Galois cover 

^ Pfe with group G = Zp X! (a) is ramified over 0, 00. The elementary abelian group E 
fixes either or c». If E fixes 00 then the arbitrary t e is of the form r : a; + c(r), 
for some c(r) S fc. In this case c is a root of the polynomial G\{x) := f{x + Ci) — f{x). 
If E fixes then we consider the model = f* (x) of the hyperelliptic curve and now E 
fixes 00. In this case c is a root of the polynomial 62(2;) := f*{x + Cj) — f*{x). 

The possible primes p, so that there is an elementary abelian E group in the reduced 
group modulo p, are the divisors of 2g + 2, if E does not fix any of the roots of the right 
hand side of the defining equation of the hyperelliptic curve. If E fixes such a root then 
p \ 2g + 1, but then there are a lot of cyclic groups in the reduced group. This is the case 
in ^(48, 7) and in A(50, 5). 

Looking at the degrees of modular hyperelliptic curves Xo{N), where N is composite 
we obtain that only for p — 3 the reduction might have some extra automorphisms. The 
modular hyperelliptic curves with degree divisible by 3 are those with N E {22, 26, 28, 50, 46}. 
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Table 7. Tables of d (TV) , G2 (^) 



22 


Gl 
G2 


(2c^ + 2c)x^ + (2c^ + c)x + c'^ + 2c-^ + 20^ 

\ J \ J 




26 


Gl 
G2 


(2 c? + c)x^ + xe" + + 
x^c^ -^2x0 + 20^ + 0^ 


28 


Gl 
G2 


(c^ + 2c)x'^ + {2c^ + c)x + 2c^ + 2c^ + 2c^ 
(c^ + 2c)x'^ + {2c^ + c)x + 2c^ + 2c^ + 2c^ 


28b 


Gl 
G2 


[c? + 2c)x'^ + {2c' + c)x + 2c^ + 2c^ + 2c^ 
{c' + 2c)x^ + {20^ + c)x + 2c^ + 2c^ + 2c^ 


28c 


Gl 
G2 


2 x^c^ + + c? 
2 x^C' + + 


50 


Gl 
G2 


(c^ + c)x^ + {c^ + c)x + 2c^ + + 2c^ 
(c^ + 2 c) + (2 + 2 c) X + 2 + 2 + c2 


46 


Gl 





37 


G2 






On the generic fibre if 7^ 28 then the reduced group on the generic fibre is isomorphic 
to Z/2Z while for = 28 the reduced group on the generic fibre is isomorphic to D^. 

We compute the polynomials Gi [N] and G2(iV) for every N e 22, 26, 28, 50, 46 and 
we arrive at Table |2l 

From that table we see that the curves Xo(22)3, Xo(28)3, Xo(50)3, Xo(46)3 admit an 
extra Z/3Z automorphism group, while Xo(26)3 does not. 

3.4. The (46) curve. The curve Xq (46) is a curve of genus 5 and has reduced automor- 
phism group in the generic fibre isomorphic to Z/2Z = (fii). We compute the following 
model of our curve, so that <Ji acts like ui : x ^ —x. 



2 12 

y ^ X 



(-4896 V2 - 6786) a;i° (-3512 x/2 - 4891) x* 2652a;6 



-2392^2 - 3381 -2392^2 - 3381 -2392 x/2 - 3381 

(3512 V2 - 4891) a;-* (4896 ^/2 - 6786) -3381 + 2392^2 
-2392^2 - 3381 -2392^2 - 3381 -2392 \/2 - 3381 ' 
We compute the decomposition of the coefficients of the above polynomial and we see that 
there is no prime P of Z[V2] with N{P) ^ 2, 3, 23 so that the reduction of the curve at 
this prime has a cyclic group containing di . 

We compute the dihedral invariants of this curve. We know that if the reduced group G 
contains (Z/2Z)-^ as a subgroup, then 2^^^u\ — u^g^^ = lfT2l th. 3.8]. The primes p such 
that 2^^^u\ — Ug^^ becomes zero modulo p are possible primes where the reduced group 
can be large enough to contain (Z/2Z)^. We compute that 



„ 9-1 
2~ui 



Ul 



223 . 312 . 3372 • 6832 



236 • (-147 + 104^2)5(147 + 104v^) ' 
210 . 312 . 72 . ^32 . 28x2 . 7092 

236 • (-147 + 104^2)5(147 + 104^2) 



Only the prime p — i gives rise to an extra automorphism modulo p. For this prime we 
have reduced group isomorphic to Z/2Z x Z/2Z and A(46, 3) = (Z/2Z)3. 

We will now see if there are primes so that ai is the involution of a dihedral group and 
there is an element t of order n so that cti , r generate a dihedral group. We consider the 

polynomials g{x) = ^y^z-jyf (irf ) ~ where A ~ ( "^^ i[ ), so that ai acts on the 

model = g{x) as an involution sending x 1 jx. The polynomial g{x) is given by 

g(x) = gi{x)g2{x), 

where 

/ 21 

giix) = fa;'' + 5/2%/2x''^ + 5a;* + — y2a;^ + 5 2;2 + 5/2%/2.T + l 

.92(2;) = ix^ + 5/2 72x5 ^ J ^4 _^ :i_ y2a;3 + 7x2 + V2x + 1 

A model of a dihedral group acting on the rational function field is given by Tnorm • 2; i-^ j 
and (Tnorm ■ X (x, whcrc C is a primitive n-th root of one. This model is conjugate by a 
Mobius transformation Q : x i-^ ff+S '^^ ^t^Y other dihedral action on the rational function 
field. We have normalized so that QrnormQ^^ = Tnorm- This means that for a, 6, c, d 
we have 6 = Ac, a = \d, d = \a, c ^ Xb for some non zero element A. This gives that 
A = ±1 therefore a ~ ±6 and b — ±c. If there is a dihedral group acting on our curve 
modulo p then there should be a transformation of Q of the form x i-^ such that 

the polynomial g'gix) has zero the coefficients ai,a7,aii of the monomials x,x^,a;^^. 
The coefficients ai, ay, an are polynomials of a, b and we can eliminate a by using the 
resultant determinant. We compute: 

^ , , , . 72 • 2310 • 608760587932768932 ,144 
Resultanta(ai, ajj = " 

The reduced curve can have more automorphisms only at the primes dividing the numerator 
of the resultant. By reducing the hyperelliptic curve modulo all that primes and studying 
its automorphism group we see that the only prime where the reduced automorphism group 
grows is p = 3. 

4. The prime iV-CASE 

If is a prime number iV 7^ 37 so that Xo{N) is hyperelliptic then on the generic fibre 
there is only one involution, the hyperelliptic involution. In order to determine the possible 
primes p 'f N so that Xo{N)p has automorphism group greater than Z/2Z we proceed 
as follows: Suppose that the curve admits the hyperelliptic model ~ fN{x) where 
fNix) £ Z[a;]. We consider an arbitrary Mobius transformation a given by a; i-^- ffip^. 
Then we consider the coefficients of the polynomial 

(6) fN{x) - In f {cx + d)^^^^" = 

If a is an automorphism then all should be zero. We would like to find if the system 
fli = has solutions modulo p. We consider the ideal :~ {ai,i — 1, . . . ,r)<jZ[a,b,c,d] 
where r < deg/^r. We compute a Grobner basis for the ideal Ir with respect of the 
lexicographical order a < b < d < c, and then we form the set S of all basis elements that 
are polynomials in c only. Since in the generic fibre the only admissible automorphism is 
the trivial one, the greatest common divisor of elements in S is c" for some 1 < q G N. 
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We divide every element in S by c" and we obtain an integer S as an element in the set 
{//c" : / G S}. The prime factors p of 5 are exactly the possible primes where an 
automorphism a with c 7^ can appear 

We do again the same procedure, but now we choose the a lexicographical order where 
a < c < d < b. We find again an integer 5' and the divisors of S' are exactly the possible 
primes where an automorphism a with d ^ can appear 

If we select a big r then the procedure of finding the Grobner basis is difficult. If on 
the other hand we select a small r (4 < r since we need at least four equations in order to 
find a unique solution in a, b, c, d) then we obtain big integers S, 5' that we are not able to 
factorize. A selection of r = 6 allows us to perform the computations needed for the given 
set of N. 

Now for each prime p that is a divisor of 5 or 5' we consider the ideals /jog /„ ®z ^/p^^ 
and we do the same elimination procedure. The Grobner basis computation is easier to 
perform over a finite field and we finally arrive at a solution of the system = mod p. 

For example, for the A'^ = 41 case the only exceptions can happen at the primes 
2, 17,41. The primes 2,41 are excluded so we focus to the p ~ 17 case. We reduce 
our curve modulo 17 and then we compute that the ideal /dcg/41 ®i Z/pZ has a Grobner 
basis of the form: 



We will now solve the above system. If 5 = then we see that c = and a = d, 
therefore we obtain the identity matrix. If 5 7^ then 6^ + 13 = 5* = 2. Let 
& be a fourth root of 2 in F17. Then c = —86, d = —8b, a = —95. The equation 
d^ + 12b^ + 16 is compatible with the system. Thus we obtain the extra automorphism a 
so that (J : X 1-^ {—9bx + b)/ {—8bx — 96) = {9x — l)/(8a; + 9). The automorphism group 
in this case is (Z/2Z) x (Z/2Z). 

There might be also an extra automorphism a modulo p so that b = c — 0. Then 
a : X 1-^ {a/d)x where a/d is an n-ih root of unity. This does not happen for any prime p 
as one easily checks. Our results are collected in table[T] 

Remark 4.1. For the case N — 71, the polynomial fjiix) has degree 14. Then the 
coefficients in ^ are polynomials of degree 14 in a, b, c, d. The computation of the 
Grobner basis over Z is very time and memory-consuming. For such situations, we use 
the following trick. If we homogenize the polynomial /at (a:) into a binary form /at (a;, y), 
then the property Q means that fN{x,y) is invariant under the substitution {x,y) t-^ 
{ax + by, cx + dy). Any transvectant 



of fN{x, y) with itself is also invariant under this substitution. (See fS] page 54].) Thus, for 
Aro(71) we first work on g — {hi, /7i)^° of degree 8. We determine all possible primes 
p such that there exists non-trivial automorphisms for g modulo p. We then determine 
whether these primes indeed give extra automorphisms for Xo(71)p. 



In this section we will study modular hyperelliptic curves Xq {N) for N odd, in charac- 
teristic 2. These curves admit a minimal Weierstrass model of the form y^ + q{x)y ~ p{x) 
ifTsl . that can easily be found with the help of magma algebra system. The equations are 



{a+16d + 



b, d^ + 12b' 



+ 16, b{d +8b),c + 8b, b{b^ + 13)}. 




5. Automorphisms in characteristic 2 
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Table 8. Global minimal Weierstrass equations for Xo{N), N odd 



JV 


Equation 


23 


+ (x^ + X + l)y = -2x'^ - 3x'^ + 2x - 2 


29 


+ (x^ + l)y = -x^ - 3a;'' + 2x^ + 2x - 2 


31 


y^ + (x^ + X + l)y = -2x^ + x'^ + Ax^ - 3x^ - 4x - 1 


33 


y^ + (x'^ + x^ + l)y = 2x^ + 9a;'' + 272!^ + 56a;* + 81a:^ + 85a:^ + 54a; + 20 


35 


y^ + (a;* + x^ + l)y = -x'' - 2x^ - x^ - 3a;* + a:^ - 2x'^ + x 


37 


y^ + (x^ + x^ + X + l)y = 31^ + Sa;" + llx^ + 8x^ + 3x 


39 


y^ + (x'^ + x^ + x^ + X + l)y = -23:'' + 2a;^ - 7a;* + 2a;^ - 2a; 


41 


y^ + (a;* + x)y = -x'' - 2x'^ + 2x^ + 5a;* + 2a!^ - 4a^^ - 5i - 2 


47 


y"^ + (a;^ + a;-'* + l)y = x"^ - Sx'' - 3ix'^ - 74x^ - 106a;* - 103a;^ - 671^ - 25a; - 5 


59 


I/^ + (a;'' + l)y = a;'' - 7a;^ - 213:** - 38a;'^ - 513;" - 53a;^ - 44a;* - 30a!^ - 17a!^ - 6a; - 3 


71 


+ (a;'' + a;" + a;* + a; + l)y = -3x^^ + 93;^^ - 17a;l'- + 16x^° - 12x^ + 
+3x^ + 9x'' - 17x^ + 16x^ - 15x* + lOa;^ - 7a:^ + a: - 2 



given in Table |5] These models are examples of the unified Artin-Schreier-Kummer theory 
in the sense of T. Sekiguchi, N. Suwa El, ll24ll . 



Lemma 5.1. Let := + q(x)y-\-y(x) he a hyperelliptic curve of genus g over F2 with 
deg q{x) < 5 + 1 and degp(a;) < 2g + 1. Then every automorphism a of^ is of the form 



ax + b y + h{x) 



^cx + d' {cx + d)3+'^ ^ 
for some (° ^) S GL(2,F2) and h{x) £ ¥2[x] of degree at most g + 1 satisfying 

{cx+dY+^ = q{x), p f (ca:+d)29+2 = p{x)+h{xf+q{x)h{x). 



In particular the hyperelliptic involution is given by 

(7) j{x)=x, j{y) = y + q{x). 

Proof. The function field of ^ is an Artin-Schreier extension of the rational function field 
F2(a;). Indeed, if we set y = y/q, then we have 

Y' + Y=P-, 
q^ 

and the hyperelliptic involution is given by (a;, Y) 1-^ {x, Y + 1), i.e, a{y) = y + q. The 
hyperelliptic involution is in the center of the automorphism group. Thus, the restriction of 
an automorphism a of '^^ to F2(.t) gives an automorphism of F2(.t). Therefore, we must 
havecr(x) = (aa; + &)/(cx + d) for some (^^) in GL(2,F2). 

Recall ifTSl prop. 1.12] that a basis for the space of holomorphic differentials on ^ is 
given by 

x^~^dx x'^^^dx 
2y + q q 

and every automorphism <t of induces a linear action on the space of holomorphic dif- 
ferentials. Write q{{ax + b)/{cx + d)){cx + = q*{x) G ¥2[x]. We find 

^ aixy-^da{x) ^ (^^d-hc){ax + br-\cx + dY-' — . 
q{a{x)) ' q* 
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Since each a{LUi) is a linear combination of ujj, we must have q* = Xq for some A e Fj. 
Because for any a e F2, a {cd) defines the same automorphism on ¥2{x) as ( ° we 
may rescale a, b, c, d so that A = 1, i.e., we have 



(8) 



We now consider a{y). We write it in the form a{y) = fiy + v with /i, £ F2(.t). 
Substituting the expression into (T(y)^ + q{a{x))(j{y) + p{(t{x)) ~ and using ([S]), we 
obtain 

2 g(a^) p(a(a:)) + ;^'+g(a:)t^/(CT + d)g+i _ 

Comparing this with y'^ +q{x)y+p{x) = 0, we find /i = l/(cx + (i)^+^ and u{cx + d)^^-^ 
is a polynomial h{x) such that 

P f^^) + ^)'''^' = Ma^) + H^)^ + q{x)h{x). 
\cx + a J 

This completes the proof of the lemma. □ 

Lemma 5.2. Let X be a hyperelliptic curve in characteristic 2. The group structure of the 
full automorphism subgroup G of X is determined by the structure of the 2-Sylow subgroup 
ofG. 

Proof. Let G denote the reduced group of G. By the theory of group extensions, the group 
G is determined uniquely by a cohomology class in the group H^{G, Z/2Z) corresponding 
to the first row of eq. |9] 

For p-prime let (G, Z/ 2Z)p denote the p-part of the finite abelian group (G, Z/2Z) 
and let Gp denote the p-Sylow subgroup of the reduced group G. The following map is 
monomorphism: 

$ : H'\G, Z/2Z) = H^{G, Z/2Z)p ^ H^{Gp, Z/2Z), 

p\s p\s 



p\s p\s 

1311 p. 93]. Now if {p, 2) = 1 then H'^{Gp, Z/2Z) = 0. Therefore we have to consider 
only the p = 2 case. This proves that H^{G, Z/2Z) is itself a group of order 2. More- 
over, the restriction map H^{G, Z/2Z) H^{G2, Z/2Z) is a monomorphism. The class 
resQ [a) corresponds to the subextension given by the second row of equation|9] 

(9) 1 *-Z/2Z 




1 Z/2Z ^ G2 ^ G2 ^ I. 

This means that the structure of G2 determines uniquely the structure of G. 

CoroUary 5.3. //G2 = {1} then G = Z/2Z x G. 



□ 



Corollary 5.4. Let G2 denote the 2-Sylow subgroup of the reduced group. This group is 
elementary abelian and fixes only one point 00 o/P^. If 00 does not ramify in X —> ¥^ 
then G^Gx Z/2Z. 
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Proof. If oo does not ramify in X ^ then there are two points Pi , P2 of X above 00. 
The group (cr) := Z/2Z = Gal(X/P^) transfers Pi to P2. Consider the isotropy supgroup 
G2{Pi)- By the transitivity of the ramification index we have that 1 6*2(^1)1 = 1^21. 
The map G2 {Pi ) ^ G2 is onto and since the two groups have the same order it is an 
isomorphism. Therefore, tt has a section and last Une of (|9]) spUts. The first fine splits also 
and G = G X Z/2Z, since the hyperelliptic involution is central. □ 



The group G2 is elementary abelian therefore is it is generated by the commuting ele- 
ments Xi, i ~ 1, . . . , s. If the group G2 is not a direct product then there are elements say 
(7 G G2 of order 4. Every such element when raised to the square gives the hyperelliptic 
involution, i.e. j ~ cr^. For the elements Xi such that 7r~^ (2;^) is a cyclic group of order 4 
select a generator Ci for this group. All these elements have squares equal to j. Order the 
elements Xi so that for 1 < i < i^o the group 7T~^{xi) is a cychc group of order 4 and for 
I'o < 1 the group tt~^{ {xi)) is a direct product of two cyclic groups. The group G2 admits 
the following presentation in terms of generators and relations: 



G2 := ( j, CTj,, a;^ : 1 < < I'D < 



1 = [Xf„Xf,'] = [x^,al] = [<T^,(J^'] 

For every Weierstrass model given in |5] we use lemma 15.11 in order to determine the 
automorphism group in characteristic 2. For every entry in that table we do not get any 
new automorphism except in the cases = 33, 37. 



5.1. CaseXo(37). By Table|5] a Weierstrass model for Xo(37)/2 is given by = 
p{x) with 

q{x) = x^ + x^ + X + 1, p{x) = x^ + x^ + X. 

The hyperelliptic involution is j : {x,y) 1-^ {x,y+q{x)). Let G denote the automorphism 
group of Ao(37) /2 and G = G/ (j) be the reduced automorphism group, considered as a 
subgroup of Aut F2 (x) = PGL(2, F2). According to Lemma ISTl an element of G takes 
the form 



ix,y) 



ax + b y + h{x) 



cx + {cx + d)^ 



where h{x) = Uq + Uix + U2x'^ + u^x^ G F2[a:;] is a polynomial of degree at most 3 and 
GGcGL(2,F2)satisfies 

\cx + dj \cx + dj 



These two conditions give a set of relations among a, 6, c, d and Ui. The Grobner basis of 
the ideal generated by these relations with respect to the lexicographic order uq > ui > 

16 



U2>U3>a>b>d>cis 
UO + US + d^c" + d'c + dc' + dc^ + ci«2 + ci«o + ci«« + ci65 + ci5° + ci38 + 

+ + cl20 + cl05 + c«6 + ^90 ^ ^84 ^ J5 ^ ^69 ^ ^66 ^ ^48 ^ ^36 ^ ^18 ^ ^9^ 

+ 7.3 + d^c + dc' + ci6« + ci3« + ci2" + ci"5 + + J5 + J2 + C«" + 

+ 7/3 + rf'c^ + dc^ + Cl8" + cl65 + cl5" + c"4 + cl35 + C^^f + c^^^ + C^^ + C^^ 

ul + U3 + d^c^ + d^c + dc'"^ + dc^ + c''^ + c^'"' + + c^* + c*' + c^ 
a + d + c^^ + c, 

d^ + d\ + dc^ + Cl«2 + + Cl32 + + + + + + 

+ + + 1, 

+ c) + + cl«l + + ^131 ^ ^80 ^ ^65 ^ ^56 ^ ^41 ^ ^26 ^ ^20 

+ + + e> + c2, 

(C^^ + C)(cl92 ^ ^144 ^ ^132 ^ ^129 ^ ^96 ^ J2 ^ ^66 ^ ^48 ^ ^36 ^ ^33 
+ c24 + cl8+cl2+c0+c6+c3 + l). 

Here the polynomial of degree 192 in c in the last element of the Grobner basis is a product 
of 12 irreducible polynomials of degree 8 over F2. Using this basis, we find that the total 
number of solutions in F2 is 480. (Each root of the degree 192 polynomial gives two 
solutions and each root of c^^ + c gives 6 solutions.) However, since for each root a of 

a;"^ + 1 in F4, (wq, mi, 1*2, U3, a, b, c, d) and (ug, ui, U2, U3, aa, ab, ac, ad) give the same 
automorphism, we find that 

|G| = 480/3 = 160, |G| = |G|/2 = 80. 

We now determine the structure of the automorphism group. 

We first consider the reduced automorphism group G. Recall that, in general, the order 
of a matrix in PGL(2, F2) can only be 2 or an odd integer. Moreover, the order is 1 or 2 if 
and only if the trace is zero. Now the relation a + d + c^^ + c = shows that an element 
(° ^) e G has order 1 or 2 if and only if c G Fig. Therefore, we find that the Sylow 
2-subgroup of G is an elementary abelian 2-group of order 16, and is normal in G. (Again, 
each c G Fie gives 3 solutions (a, 6, c, d), but for each root a of a;'^ + 1 in F2, (a, 6, c, d) 
and (aa, a6, ac, ad) correspond to the same reduced automorphism in G.) The remaining 
64 elements of G all have order 5, and G has 16 Sylow 5-subgroups. Therefore, G is the 
semi-direct product of an elementary abelian 2-group of order 16 by a cyclic group of order 
5. 

Now consider the structure of G itself. Let P be its Sylow 2-subgroup, and t be any 
element of order 5. The centralizer Zp{t) of t in P must satisfy |Zp(t)| = |P| = 
32 mod 5. Thus, we have |Zp(t)| = 2 or |Zp(t)| = 32. The latter possibility cannot 
occur as it would imply that G is an abelian group. Thus, we have |.Zp(r)| = 2, that is, 
Zp{t) = (j). 

We next turn the attention to the center Z{P) of the Sylow 2-subgroup P itself. Observe 
that (r) acts on Z{P) by conjugation. The identity automorphism and the hyperelliptic 
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involution are left fixed by this group action. Since |Zp(r)| = 2, all the other orbits 
under this group action have 5 elements. In other words, we have |^(P)| = 2 mod 5, i.e., 
|Z(P)|=2or|Z(P)H32. 

Assume that |2^(-P)| = 32. Then P is an abelian group of order 32 whose elements 
have order at most 4 (since P/ (j) is elementary abeUan). Noticing that t acts on the set 
of elements of order 4 in P and that |Zp(r)| = 2, we see that the number of elements of 
order 4 in P must be a multiple of 5. The only possibility is that P ~ (Z/2Z)^. However, 
we can easily check that the automorphism a E G given by 

a : {x,y) ^ \ax + [a + 1)' [ax + [a + l)f^ 

is an element of order 4, where a is a root of + a:; + 1 in F4. Therefore, we conclude 
that \Z{P)\ cannot be 32. Instead, we have \Z{P)\ = 2, i.e., Z{P) = {j). 

Now we have |^(P)| = 2 and P/Z{P) is elementary abelian. This means that P is 
one of the extraspecial groups. (An extraspecial group H is a p-group such that the center 
Z is cyclic of order p and the quotient group H/Z is a non-trivial elementary abelian p- 
group.) For order 32, there are two extraspecial groups E32+ (£'4 x P'4)/((a, a)) and 
i?32_ := {D4 X Qs)/{{a, b)), where a and b denote the non-trivial elements in the centers 
of the dihedral group D4 and the quaternion group Qs, respectively. To determine which 
one P is isomorphic to, we consider the action of (t) defined by conjugation on the set S 
of subgroups of order 16 in P. 

We claim that tAt^^ ^ A for all A e S. Assume that tAt^^ ~ A for some A e 5. 
The centralizer Za(j) of t in A must have only one element since |Zp(t)| = 2 and 
|Zyi(r)| = \A\ mod 5. In other words, j ^ A, but this would imply that P ~ A x 
(j), which cannot be true for an extraspecial group. Therefore, we must have tAt^^ ^ 
A. It follows that for each group H of order 16, the number of subgroups of P that are 
isomorphic to H must be divisible by 5. Now according to the database of small groups 
ifTOl . i?32+ has 9 subgroups isomorphic to D4 x (Z/2Z) and 6 subgroups isomorphic 
to another group Hiq of order 16. Therefore, we conclude that P must be isomorphic 
to E32-, which has 5 subgroups isomorphic to Qs x (Z/2Z) and another 5 subgroup 
isomorphic to Hie, and the automorphism group G is a semi-direct product of by a 
cyclic group of order 5. 

Of course, the conclusion above can be verified by brute force computation. However, 
the computation is too complicated to be presented here. 

5.2. Case Xq{33). The conditions given in lemma ISTI give rise to a system in a, b, c, d 
describing every element in the reduced group G. The Grobner basis of this system is 
computed to be: 

+ ac + + dc, ab + ac + bd + dc, ad + be + d"^ + dc + , ac^ + a + dc^ + d, 

b'^ + bd + dc + c^, bdc + bc^ + dcJ^ + c^, d'' + dc^ + c'' + 1, d^c + dc^, + c 
This gives us the following solutions (written in matrix form): 

„ , /I Q\ fO l\ fO l\ (\ \\ (\ \\ fl 0^ 



ly'Vi oy'^i i/'Vo i/'Vi i 

Thus, the group G is isomorphic to the group GL(2, F2) of order 6. One 2-Sylow subgroup 
of G is given by the group generated by the element t : x ^ x + 1. The fixed point of t 
is the point 00 and since 00 is not ramified in the cover X Corollary El impUes 

thatG = G X (j). 
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